@trentapple thanks for sharing your changes. I applied them to #91 along with #87. However, unfortunately the e2e tests are still failing (see here). I guess you didn't make them work either, did you?

Turns out the rootful networking problem was solved within podman 5.1.1 - and so was the port mapping, except for my local machine where port mapping still does not work for some reason.
Now the only remaining e2e test that fails within my CI build is the one calling podman play kube.

Using --network host still works locally though I presume? (Different from port mapping — of course). Rootful, rootless, or both when local? With either pasta or slirp4netns? port_handler or pasta options?

My initial thought before reading any of this was nftables (I recall the netavark team had initially integrated libnftables support around the beginning of the year). Although that may not matter depending on the scenario.

Out of curiosity, any digging into some lower level details with (net) namespaces, kernel parameters, cgroups, logs, or anything along those lines (aside from kernel version) that may differ between CI and local?

I did not specifically get the e2e tests working either, but it is great to hear initial reports of an issue for rootful being fixed in 5.1.1.

So the remaining e2e test is the following (if I understand correctly): test/rootless.bats#L58

Recent job run #109 output

not ok 11 rootless podman - play kube in 649ms
# (in test file test/rootless.bats, line 63)
...
make: *** [Makefile:78: test-use-cases] Error 1
Error: Process completed with exit code 2.

Just based on the output it seems to be complaining about rootless netns: mount and no such file or directory found. Occurs when mounting /tmp/storage-run-1000/containers/networks/rootless-netns/run to /run as you’ve determined.

Now seems that upstream libnetwork code in podman relied on a relative path existing which was created implicitly for systemd setups.

I did not specifically get the e2e tests working either

A pity.

Using --network host still works locally though I presume?

Sure, but the goal is not to make some feature work but to make all the common container engine use-cases work and thus to make the e2e tests that worked with podman v4 also work with podman v5.
However, the port mapping issue isn't critical since it works within the CI pipeline already and I also think the problem (why it does not work on my local machine) might be related to nftables, indeed. I'll have to dig a bit deeper into that.

Out of curiosity, any digging into some lower level details with (net) namespaces, kernel parameters, cgroups, logs, or anything along those lines (aside from kernel version) that may differ between CI and local?

No, at this point I didn't dig deeper yet and I'd prefer solving the issue with logic on a higher level 😄.

The podman play kube test is more urgent to fix since it blocks the release within this repo.

So the remaining e2e test is the following (if I understand correctly): test/rootless.bats#L58

Exactly.

Now seems that upstream libnetwork code in podman relied on a relative path existing which was created implicitly for systemd setups.

I didn't see that but I also suspect that something may be creating that directory relative to one that is obtained from an env var or so that is missing within my image - to be investigated. It doesn't seem to be related to a systemd setup since the test scenario works with the official podman image which creates the /tmp/storage-run-1000/containers/networks/rootless-netns directory correctly when podman play kube is called. To be investigated...

There may be a nicer location (and not even 100% certain the directory will be created in the right place there), but feel free to try #92 if you wish as a way to create the dir before test. This will only be necessary until the patch from upstream makes it into a release as far as I understand it (to be confirmed of course).

Ah, after reading the response within the corresponding discussion I realized that you were right about the systemd integration (that is missing within my image) is the causing the rootless-netns directory to not exist.